AFRL-RX-WP-TP-2011-4401 



DEMONSTRATION STUDY FOR RELIABILITY 
ASSESSMENT OF SHM SYSTEMS INCORPORATING 
MODEL-ASSISTED PROBABILITY OF DETECTION 
APPROACH (PREPRINT) 


Eric A. Lindgren, Charles F. Buynak, and Jeremy S. Knopp 


Nondestructive Evaluation Branch (AFRL/RXLP) 


John C. Aldrin 
Computational Tools 


Enrique A. Medina 

Science Applications International Corporation 

Jose Santiago 
Consultant 


NOVEMBER 2011 


Approved for public release; distribution unlimited. 

See additional restrictions described on inside pages 


STINFO COPY 


AIR FORCE RESEARCH LABORATORY 
MATERIALS AND MANUFACTURING DIRECTORATE 
WRIGHT-PATTERSON AIR FORCE BASE, OH 45433-7750 
AIR FORCE MATERIEL COMMAND 
UNITED STATES AIR FORCE 







REPORT DOCUMENTATION PAGE 


Form Approved 
OMB No. 0704-0188 


The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, searching existing data 
sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of 
information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson 
Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a 
collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YY) 2. REPORT TYPE 3. DATES COVERED (From - To) 

November 2011 _ Technical Paner _ 1 August 2011-1 August 2011 _ 













DEMONSTRATION STUDY FOR RELIABILITY ASSESSMENT OF 
SHM SYSTEMS INCORPORATING MODEL-ASSISTED 
PROBABILITY OF DETECTION APPROACH 

John C. Aldrin 2 , Enrique A. Medina 3 , Jose Santiago 4 , Eric A. Lindgren 1 , Charles F. Buynak 1 , 

Jeremy S. Knopp 1 

’Air Force Research Laboratory, Wright-Patterson AFB, OH 45433, USA 
Computational Tools, Gurnee, IL 60031, USA 

3 Science Applications International Corporation, Beavercreek, OH 45434, USA 
Consultant, Aventura, FL 33160, USA 

Abstract. This paper presents the results of a demonstration featuring the application of a validation 
protocol to a vibration-based structural damage sensing system. The results of the full validation study 
highlight the general protocol feasibility, emphasize the importance of evaluating key application 
characteristics prior to the POD study, and demonstrate an approach to quantify varying sensor durability 
on the POD performance. Challenges remain to properly address long time-scale effects with accelerated 
testing and large testing requirements due to the independence of the inspection of each flaw location. 

Keywords: model-assisted POD evaluation, probability of detection (POD), reliability, structural health 
monitoring 
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INTRODUCTION 

The successful deployment of systems for health monitoring of structures depends on 
appropriate verification and validation (Y&V) of these structural health monitoring (SHM) 
systems. The V&V method must explicitly evaluate all aspects of the SHM system that can 
affect its capability to detect, localize, or characterize damage. Moreover, it must evaluate 
the effects that usage and environmental conditions have on these capabilities over time. 
The current U.S. Air Force practice for maintaining aircraft structures follows the Aircraft 
Structural Integrity Program (ASIP) methods, as documented in MIL-STD-1530C [1]. A 
critical part of the damage tolerance approach is the assessment of the reliability of 
Nondestructive Evaluation (NDE) methods that are used for the periodic inspection of 
structures. MIL-HDBK-1823A provides guidance on probabilistic methods for NDE 
reliability assessment and introduces the use of models to complement experimentation for 
a probability of detection (POD) determination [2], As SHM methods depending on 
permanent, on-board mounted damage sensing systems continue to be proposed and 
developed for complementing ground-based NDE inspections for aircraft structural 
integrity purposes, it is necessary that the reliability of these damage sensing systems be 
assessed with a rigor that is suitable and sufficient for the function that they are expected to 
perform within the ASIP methodology [3], For damage detection, this necessarily results in 
the need for a POD determination. For localization and characterization, the metrics and 
their evaluation process are the subject of current research and development [4], This paper 
will present a demonstration of the damage detection protocol utilizing empirical data, 
models, and uncertainty analyses for characterizing SHM reliability [5], 
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EXPERIMENTAL STUDY SETUP 


The example used for this initial demonstration of the protocol is a system for detecting 
the presence of damage using permanently mounted transducers. A test article representing 
an aircraft structure of medium complexity was designed and built. The test article consists 
of three plates connected by two lap joints with fasteners (see Figure 1). In addition, a 
fixture was built for supporting the test article. Fatigue crack damage around the fastener 
holes can be simulated by manually created thin cuts at selected locations. The test fixture 
design provides the capability to vary critical parameters of the system with a focus on 
force loading boundary conditions, joint fastener torque conditions, and temperature. 

The initial demonstration on this test article and fixture uses a vibration based damage 
detection method. An ETrema brand Terfenol-D magnetostrictive actuator was used for 
band-limited pseudo-random excitation up to 1200 Hz, and the dynamic response of the 
plate was recorded using eight 50mV/g single-axis accelerometers set to measure out-of- 
plane motion and provide input to change detection algorithms. The accelerometers were 
placed at locations that attempt to maximize the ability to detect change in the modal 
dynamics of the structure even in the presence of modeling errors. M-Bond 200 adhesive 
was used for semi-permanently affixing the accelerometers to the bottom of the assembly. 
Four bonded foil 350-ohm strain gages were also installed on the test article for tracking 
the state of stress under mechanical loading. Finally, thermocouples were also used for 
studying the effects of heating due to excitation of the damage detection system and the 
effects of the operational temperature. The layout of the sensors is presented in Figure 1. A 
Lab VIEW data acquisition system was used for acquiring the required data. Variations in 
operational temperature were simulated by testing the system inside a carefully controlled 
Thermotron SE-1200 environmental chamber. 

The method for inducing damage involved notching the area of the „skin" under a 
fastener. To initially isolate damage to the skin alone, the test article design includes 
machined relief slots in the joining plate at the fastener locations. A change metric based on 
the area under the curve representing the difference between two frequency responses from 
200 to 1200 Hz and an R-square metric were used for assessing changes in structural 
dynamics due to mechanical and thermal loading, actual damage, and combinations 
thereof, and therefore attempting to detect damage presence and growth. 
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Figure 1. Location of sensors on the test article. 
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FACTOR EVALUATION STUDIES 


Following the protocol introduced in reference [5], prior to designing the validation test 
matrix, the following factors were assessed through controlled studies: (a) mass loading 
and unloading, (b) fastener torque, (c) boundary condition variation, (d) temperature 
variation and temperature gradients, (e) sensor bond quality, (f) ambient noise, and (g) flaw 
growth. More details on the factor evaluation studies are presented in reference [6]. 

Thermal loading studies were performed by varying the ambient temperature from - 
20°F to 150°F. During this study, the thermal capacity of the panel end conditions fixtures 
was found to produce significant thermal gradients across the test article. During heating 
and cooling periods, temperature gradients as high as 45 °F across the test specimen were 
observed. For validation studies, an estimate of expected gradients „inthe field" is needed. 
An assumption was made for the validation study that temperature gradients in the region 
of interest will be primarily limited to +/- 10°F. 

To address the temperature conditions, a new temperature compensation algorithm was 
designed and implemented that (1) uses multiple FRF references at different temperature 
bands and (2) incorporates a nonlinear frequency shift model for local temperature 
compensation. An approach was tested that uses three baselines, one at the nominal 
temperature, one at a higher temperature range (with minimal temperature gradients) and 
one at the low temperature range. The algorithm then evaluates the minimum of the three 
damage metric results. A two parameter frequency shift model was introduced 
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where </>o and (j)\ represent bias and slope adjustments, respectively. A nonlinear least 
squares estimation routine was implemented to evaluate <jx) and (j)\ and a log transform of 
the FRFs was used to enable sensitivity to changes all along the FRF, not just at the peaks. 
Through the use of this temperature compensation algorithm, significant improvements to 
the damage metric were achieved. One downside with the algorithm is that the fit and 
interpolation process take longer (seconds) to run. As well, there is still some difficulty in 
compensating for rapid temperature changes that produce severe gradients across the plate. 

Failure of accelerometer bonding was observed several times during factor thermal 
testing. These failures occurred during the prolonged high temperature runs at 150°F. 
Coherence was shown to be a viable metric to monitor bond quality over time. While 
vibration-based damage detection systems are proposed as global methods with some 
sensor redundancy, relying on a single reference sensor will result, upon bond degradation, 
in either highly degraded performance and/or complete failure of the damage detection 
system well before the structure end-of-life is reached. Sensor and sensor bonding 
reliability must therefore be accurately assessed as part of a validation study. 

The noise generated by the environmental chamber and by other equipment in the 
laboratory posed an opportunity for studying the effects of ambient noise on the detection 
capability. Coherence levels were observed to change depending on whether the unit was 
cooling or heating and the „thiottle" level of the chamber, and on whether the chamber door 
was open or closed. For the validation study, both „chamber on" and „diamber off' 
conditions will be acquired in order to include in the analysis the effects of varying noise 
conditions that are expected in the field. 

An initial study on the effect of damage growth was performed to ensure adequate 
sensitivity during the final validation study. Customized XActo™ blades were used to 
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make cuts in the aluminum plates. The resulting notch width was found to be 0.012" +/- 
0.002". Cuts were initially made at 0.063" (1/16") increments up to 0.63". For the first 
series of cuts up to 0.25", sensitivity to notch length increases was observed, but the trend 
was small relative to noise, and not quite linear. Greater sensitivity to the larger cuts was 
observed and clear sensitivity to notches on the order of 0.63" was demonstrated. Note, a 
significant increase in the damage metric was observed after a two week delay between the 
end of the 0.25" notch cut and the start of the 0.31" notch cut. Relaxation of the boundary 
conditions over time was thought to be the source of the change. For validation, controlled 
time delays should be included into such studies to isolate and address long-time effects. 

DESIGN OF VALIDATION STUDY 

The validation study consisted of growing flaws by artificially cutting the structure at 
two fastener site locations, site 2 and site 3, as shown in Figure 1. A series of 
environmental and boundary conditions were studied after each flaw growth scenario: 
temperature variation (+/- 40°F), temperature gradients, loading and unloading of 10 lb 
mass, a simulated maintenance action at a set of fasteners (see Figure 1) including the case 
of minor loosening, and reinstallation and replacement of accelerometers. During the flaw 
growth period, tests were performed before and after flaw growth, and after fastener 
installation. Temperature chamber states of „on"and „off'with the ambient temperature set 
to T = 72°F were acquired for all test conditions. Five averages were taken for all SHM 
system acquisitions. After any environmental condition or change to the test fixture, testing 
was performed with the ambient temperature returned to T = 72°F. 

POD ANALYSIS APPROACH 

A summary description of the approach being followed for obtaining a probability of 
detection model for on-board damage detection cases follows. Conventional probability of 
detection (POD) evaluation for many quantitative NDE applications first uses empirical 
data to evaluate statistical relationships between the measurement response,a, and the 
primary flaw size variable, a. Through application of a detection criterion as part of the 
NDE procedure, this statistical „a versus a’ model can be used for evaluating the POD 
curve and probability of false call (POFC) rate, which together are usually referred to as “a 
POD model”. The detection system can be abstractly represented by a set of random 
variables a t that act as inputs to a measurement model. Input variables can be categorized 

as being controlled (e.g. flaw size and material properties) or uncontrolled (e.g. liftoff, flaw 
morphology, and measurement noise). Detection consists of the measurement model output 
a being classified (or “called”) according to pre-specified rules (e.g. a threshold). 

The model-assisted POD (MAPOD) approach proposes to replace a conventional 
statistical fit in the measurement model with a complete physics-based model,/, calibrated 
for a given set of experimental conditions. This relationship is given by 

& = Pq + A/( a ; ) + £ ■> (2) 

where (3 ,o and (3\ represent the model calibration parameters, and s represents the residual 
error between the model and the experimental data. Estimating the statistics of /?o, fi\, and s 
necessitates specific experimental sampling requirements. Variations due to flaw size and 
environmental (noise) conditions, for example, are represented in the model as probability 
distributions of the input variables. Hybrid models incorporating both empirical and 
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physics-based components can be implemented to address all key factors including those 
that cannot be adequately simulated. For this study, due to a lack of a validated physics- 
based model, a surrogate model fit using empirical data was applied in the evaluation. 

In this study, the primary variable associated with the critical flaw size is crack (notch) 
length, a\. Controlled secondary variables in the study include flaw location (a 2 ), mean 
temperature (< 23 ), temperature gradients (< 24 ), ambient noise level (a 5 ). A response surface 
methodology was applied here to estimate the effect of each factor on the damage metric 
response and construct a model,/(a.) including uncertainty. Random events such as sensor 

failure / disbond ( b \), sensor bond degradation (hi), sensor replacement (£ 3 ), and local 
maintenance actions (Zq) were considered in the POD evaluation study. Assumptions 
concerning their frequency can be made and empirical models representing their effect can 
be evaluated and applied in conjunction with the scope of the SHM application. 

To complete the POD evaluation, an assessment of the detection model under varying 
input conditions including uncertainty propagation is necessary. In [ 8 ], a second-order 
probabilistic approach has been developed to propagate both aleatory uncertainty, due to 
inherent randomness in system behavior, and epistemic uncertainty, due to a lack of 
knowledge about values expected to be fixed. Using this approach, epistemic variables are 
specified as intervals on values of parameters such as the means and standard deviations of 
random variables. Monte Carlo analysis is applied here using outer and inner loops. The 
outer loop varies the values of distribution parameters of selected epistemic variables while 
the inner loop samples from the distributions. For this study, the distributions for the input 
variables, mean temperature and temperature gradients, are presented in Figure 2. 

POD EVALUATION RESULTS 

Sensitivity of POD to Flaw Location 

Following acquisition of the experimental data, a regression model fit was performed to 
evaluate f(a i ) using the R software environment. Three different flaw models were 

considered in the evaluation: (a) a flaw 2 and 3 combined evaluation (including flaw 
location factor, a 2 ), (b) a flaw 2 evaluation only, and (c) a flaw 3 evaluation only. One 
reason for performing and studying separate model fits for the different flaw growth sites 
was due to early observations that the SHM system was more sensitive to changes in flaw 3 
with respect to changes in flaw 2. POD analysis results for the vibration-based SHM study 
are presented in Figure 3 with respect to flaw size (in inches) for the case of a damage call 
threshold of 0.05. The damage metric initially tested was based on taking the median 
response from all of the active frequency response functions (FRF), in order to minimize 




Figure 2. Distributions for temperature parameters: (a) mean normalized temperature ( T), (b) maximum 

temperature gradient (dTmax / 10°F). 
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sensitivity to outlier sensor responses during operation. For each POD evaluation, both 
input parameter variation and model uncertainty are addressed through a two-level Monte 
Carlo simulation. To minimize simulation time, 2000 samples were used for each level, and 
the POD evaluation was performed at 51 different flaw sizes from 0.0 to 1.0 inch. 

From these results, there is clear need to separately evaluate the POD models for flaw 2 
and flaw 3 locations. A single POD curve does not properly address the poor detection 
capability at the flaw 2 location as a function of performance at large flaw size. Using the 
‘flaw 2 and 3 combined’ results will give one a false sense of security in terms of detection 
capability. Note, for any future SHM validation study, care must be taken to ensure the 
‘overall’ POD capability evaluations do not mask ‘isolated’ flaw locations that have poor 
detection capability. Likewise, the low false call rate in the flaw 3 model is likely due to 
the flaw 3 model only including a portion of the simulation study variation. In future work, 
the false call model should try to include all experimental study data. Lastly, POD results 
for flaw 2 shown in Figure 3(b) can be improved by using only the optimal sensor #6 data. 

Impact of Sensor Degradation over Time 

This analysis approach enables the evaluation of the impact of sensor durability on 
POD performance. To perform this study, a strategy to address sensor durability issues 
must first be clearly defined upfront, as shown in Figure 4. Two sensor failure scenarios are 
of particular interest here: failure of the optimal detection sensor (accelerometer 6), and 
failure of the reference sensor. For the example of ‘flaw 3’ results shown in Figure 4, it is 
clear how the corresponding POD curves are degraded with respect to the optimum case. 

Next, models for sensor performance degradation (i.e. failure rates) are required for the 
evaluation. To represent the failure rate of in-situ sensors like strain gauges or 
accelerometers, a ‘bathtub’ curve is often used [8]. It describes a particular form of a 
hazard function which comprises three parts: an initial decreasing failure rate, known as 
infant failures, a second part representing a constant failure rate, known as random failures, 
and a final part representing an increasing failure rate, known as wear-out or fatigue 
failures. Evidence from strain gauge sensor data on C-17 aircraft demonstrates the need for 
assessing the impact of degradation, where 22% of the sensors were infant failures and 
about 40% of the total failed within the first ten years of the aircraft life [9]. Given that 
only eight sensors are present in the subject SHM system, the scenario of 25% failures, two 
accelerometers, will be considered during the first 6 year period of operation. Figure 5(a) 
presents two probability density functions for the time to failure for the first and second 
sensors. Here, gamma distributions are applied to represent portions of the ‘bathtub’ curve. 

Data tables were constructed evaluating POD models for all of the ‘single sensor’ and 
'two sensor’ failure scenarios. A Monte Carlo simulation was then performed using 10000 
samples from the time to failure distributions for the first and second sensors. Equal 
probability of failure was assumed for the eight accelerometer locations. The POD(t) curve 



Figure 3. POD results with respect to flaw size including uncertainty bounds for (a) combined flaw 2 and 3, 
(b) flaw 2 only, and (c) flaw 3 only ( dm_threshold = 0.05, use median sensor response). 
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was evaluated at 0.1 year time increments for 6 years. Results are presented in Figure 5(b) 
for the mean value from the composite Monte Carlo simulation POD results at a flaw size 
of 1.0 inch as a function of time. Results for both flaw 2 and flaw 3 locations are presented. 
This analysis is useful because it highlights the sensitivity of certain flaw locations to 
degradation in the SHM system. In particular, the detection of flaw 2 suffers from weak 
crack sensitivity with respect to significant noise sensitivity due to varying temperature 
conditions. Flaw 3 detection capability was found to be quite robust to the failure of only 
one or two accelerometers. 


SUMMARY 


This work has presented the results of a demonstration featuring the application of a 
proposed validation protocol to a vibration-based structural damage sensing system. The 
design and results of the full validation study highlight the general protocol feasibility, 
emphasize the importance of evaluating the key application characteristics prior to the POD 
study, and demonstrate an approach to quantify varying sensor durability on the POD 
performance. However, challenges remain, in particular on how to properly address long 
time-scale effects with accelerated testing and how to address large testing requirements 
given the independence of each flaw location in the evaluation. Going forward, by better 
leveraging validated numerical models, it may be feasible to address these challenges. 
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Figure 5. (a) Case study probability density functions for the time to failure for the first sensor and second sensor, 
(b) mean expected probability of detection (POD) at a flaw size of 1.0 in. with respect to time for all SHM systems 
found in the field. 
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